Multidimensional virtual learning audio programming system and method

ABSTRACT

A process and system for generating three dimensional audio for television broadcast includes generating a virtual map of participants with a plurality of positions, each participant selecting one of the positions, determining a direction from each position to each other position on the map and to predetermined listener position, receiving sound from each participant, converting the received sound according to the direction of the speaking participant to the listener, mixing the converted sounds, transforming the mixed sound into binaural audio, and directing the binaural audio sound for transmission to a television viewer. The result is a clarified sound that gives to the television viewer a sense of where the speaking participant is positioned relative to the listening television viewer.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-provisional PatentApplication Ser. No. 14/460,537, filed Aug. 15, 2014, entitledMultidimensional Virtual Learning Audio Programming System and Method,now U.S. Pat. No. 9,197,755, which claims the benefit of priority ofU.S. Provisional Patent Application Ser. No. 61/872,068, filed Aug. 30,2013, and application Ser. No. 61/886,515, filed Oct. 3, 2013, bothentitled Multidimensional Virtual Learning System and Method, thecontents of which are incorporated herein by reference in theirentirety.

BACKGROUND OF THE DISCLOSURE

Teleconferencing, conferencing, and distance learning systems share asimilar drawback: the inability for participants to distinguish andunderstand multiple voices speaking simultaneously. Teleconferencing isa popular method of communication between multiple people. During ateleconference it is difficult to have conversations in which more thanone person speaks. This is caused by the way existing teleconferencingsystems mix together the voices of multiple speakers. Distance learningsystems, such as webinars and virtual classrooms, also have the sameissue. While distance learning systems involving a virtual classroom areknown, there is no way for more than one person to speak at a time inwhich a listener can readily differentiate between speakers.Furthermore, the entire experience is relatively one dimensional. Whatis needed is an enhanced virtual learning system in which theparticipant can feel he or she is really experiencing an actualclassroom environment with each user or participant having the abilityto distinguish between multiple voices.

SUMMARY OF THE DISCLOSURE

The present disclosure directly addresses this problem. In oneembodiment of the present disclosure, in which a person talks to anotheruser of the system, the words spoken and heard by the user are not fromjust a disembodied voice but from the person at a predefined location,for example, sitting right next to the user in the virtual classroom,webinar, or conference. Thus the system in accordance with the presentdisclosure involves a novel 3D Sound Conferencing system. A 3D SoundConferencing system makes it possible to have a conversation in whichmore than one person speaks at the same time by restoring the sound cuespresent in real life. In particular, each person in a 3D Soundconference is associated with a position in a map of a virtual room.This room can be used in a teleconference, webinar, electronicconference, electronic chat room, virtual classroom, or any groupmeeting where there is sound. The sound is then transformed so that eachperson in the virtual room hears the sound of the other people as iftheir voices originated from their specific location in the virtualroom. In this way the direction information in the sound allows humansto more easily distinguish one voice from another. Thus, if multiplepeople speak at the same time, an individual can distinguish thedifferent voices and directions of each voice. This allows a groupconversation to occur electronically in a manner similar to real life,and thereby enhances the experience in such an environment.

Throughout this specification, reference is made to a conferenceparticipant, a plurality of participants, etc. It is to be understoodthat a participant may be a listening participant and/or a speakingparticipant. In addition, reference is made to a conference, conferencesand conferencing and it is to be understood that a conference is anyform of a conference communication, including but not limited totelecommunications, conference calls, virtual classrooms, webinars,electronic group meetings, and combinations of conference communicationforms. Furthermore, it is to be understood that a conference may becomprised of n participants, where n represents any number.

One embodiment in accordance with the present disclosure is amultidimensional sound conferencing method. This method includesoperations of: generating a map of a virtual room having a plurality ofpredefined positions; determining a direction in the virtual room fromeach predefined position to each other predefined position in thevirtual room; assigning or associating a conference participant to eachof the positions on the map; assigning a virtual speaker associated witheach position; receiving sound from a speaking one of the participants;converting the voice sound to a converted sound corresponding to each ofthe predefined positions such that the converted sound corresponds toits direction from the one of the positions assigned to the speaking oneof the participants directing the sound to the virtual speakerassociated with the speaking participant's position on the map; andtransforming the sound directed to the virtual speaker to binaural audiosound. This virtual map may include a sound ring or “soundring” aroundthe positions. In such an embodiment, each virtual speaker is associatedwith a position around the sound ring.

In one embodiment the virtual room may have a plurality of wallsdefining the room. These walls may facilitate introducing reverberation,or reverb, into the sound transmitted to each virtual speaker at eachposition around the sound ring. The amount of reverb may be determinedfrom the incident and reflection angles of sound transmitted from thespeaking participant's position in the virtual room against an objectsuch as another person, a chair, or one or more of the walls definingthe room to a particular listening participant.

A method for simulating a three dimensional audio experience during aconference between a plurality of participants, in one embodiment,includes: receiving a plurality of voices; associating each voice to aunique participant; presenting to each unique participant a virtual mapof a virtual room showing a plurality of different positions in the roomequal to or greater than the number of unique participants; eachparticipant selecting a different position on the map within the virtualroom; modifying each voice according to its position on the map into amodified voice; and transmitting the modified voice to each of the otherparticipants. The method further includes determining a direction fromeach position in the room to each other position in the virtual room andassociating a different speaker with each different position in thevirtual room. Each modified voice may preferably be determined from thedirection of the speaker associated with its position in the room.

Another embodiment of the disclosure is a method for simulating threedimensional audio experiences in a conference. This method includesgenerating a map of a virtual room having a plurality of differentpredetermined positions on the map, presenting the map to a plurality ofconference participants, and either having each participant select oneof the different positions or assigning a different one of the positionsto each participant. The system then receives a voice from a speakingone of the plurality of participants. The received voice is thenmodified according to the selected position of the speaking one of theplurality of participants and then the modified voice is transmitted toeach other participant according to the direction of each otherparticipant from the selected position of the speaking participant. Themethod may also include assigning a virtual speaker to eachpredetermined different position on the map and transmitting the voicefrom the speaking participant to each of the other participants from thevirtual speaker assigned to the speaking participant.

An embodiment of the method of generating three dimensional soundconferencing in accordance with the present disclosure can includegenerating a map with a plurality of positions, each participantselecting one of the positions, determining a direction from eachposition to each other position on the map, determining a distance fromeach position to each other position on the map, receiving sound fromeach participant, mixing the received sound in accordance with thespeaker's selected position, transforming the mixed sound into binauralaudio, and directing the binaural audio sound to each participant via avirtual speaker associated with the position of the speakingparticipant.

Another embodiment in accordance with the present disclosure is a methodof and system for audio programming for television broadcast ortransmission to remote viewers of a meeting or conference between aplurality of speaking participants. In such situations, the speakingparticipants may talk over one another making it difficult or a viewerto differentiate between the speaking parties. The method in accordancewith the present disclosure solves this problem. The method includesassigning each speaking participant to a different position on a map;determining a direction on the map of each speaking participant from ahypothetical listener also located on the map; receiving a voice soundfrom the speaking participant; converting the received voice sound to aconverted sound corresponding to the direction to the listener from thespeaking participant; and transforming the converted sound to binauralsound for audio transmission. The converted and transformed binauralsound may be coupled to corresponding video and transmitted or broadcastin a conventional manner to viewers. The viewers hear the speakingparticipants as if they are within the same location as the speakers,even though they may be speaking at the same time since theseparation/distance ques have been reintroduced into the audio streamprior to transmission in accordance with the present disclosure.

Further features, advantages and characteristics of the embodiments ofthis disclosure will be apparent from reading the following detaileddescription when taken in conjunction with the drawing figures.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is flowchart of the 3D Sound Conferencing process for a flat roomwith no acoustic effects from changes in elevation and no reverberation.

FIG. 2 is a diagram of a representative small conferences room showingdirectional sound without the effects of changes in elevation orreverberation.

FIG. 3 is a sound-ring with directional sounds and virtual speakers.

FIG. 4A is a 2.5D map, a two dimensional map with some 3Dcharacteristics, of a conference room showing the direct andreverberated paths of a sound.

FIG. 4B is a 2.5D map, a two dimensional map with some 3Dcharacteristics, of a conference room showing the direct andreverberated paths of a sound.

FIG. 5A is a 2.5D conference room map with simplified reverberation.

FIG. 5B is a 2.5D conference room map with simplified reverberation.

FIG. 6 is a sound helmet with directional sounds and virtual speakers.

FIG. 7 is a flowchart of the 3D Sound Conferencing processes.

FIG. 8 is a diagram of a representative 100 seat hall where seats havebeen grouped into blocks and blocks have been grouped into superblocks.

FIG. 9 shows an example of a schematic diagram illustrating a clientdevice in accordance with an embodiment of the present disclosure.

FIG. 10 is a block diagram illustrating an internal architecture of acomputer utilized in accordance with an embodiment of the presentdisclosure.

FIG. 11a is a diagram of a typical real life panel discussion configuredfor collaboration for television broadcast.

FIG. 11b is a virtual meeting room configuration for a television paneldiscussion shown in FIG. 11 a.

FIG. 11c is a tiled configuration of the virtual meeting room shown inFIG. 11 b.

FIG. 12 is a mixer configuration with 3D Sound Positioners for atelevision panel discussion in accordance with the present disclosure.

FIG. 13 is a flowchart of the television broadcast 3D Sound conferencingprocesses in accordance with the present disclosure.

FIG. 14 is a diagram of a 3D sound positioner interface for theconfiguration shown in FIG. 11.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the description. It will be apparent, however, to oneskilled in the art that the invention can be practiced without thesespecific details. In other instances, structures and devices are shownin block diagram form in order to avoid obscuring the description.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the invention. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not other embodiments. Throughout thedescription that follows, reference will be made to a speakingparticipant and a listening participant. Each participant may be eithera speaking or a listening participant depending on what the participantis doing at the moment. In addition, even when a participant isspeaking, it should be understood that he or she can be concurrentlylistening.

Concisely, 3D Sound is sound which contains cues that convince thelistener that the source of a sound is in a specific location,direction, and distance from this listener. 3D Sound differs fromsurround sound in that surround sound just tries to surround you withsound but does not, in general, accurately recreate specific location,direction, and distance. The term 3D sound refers to the fact that mostsurround sound is limited to surrounding you with sounds seeming tooriginal from a two dimensional plane, disc, or ring around your head,whereas 3D sounds can seem to originate from any location, direction,and distance in three dimensions, such as a sphere, ball, or helmetaround your head.

Technically, commercial software uses 3D Sound to refer to machinegenerated binaural audio. In binaural audio, a pair of microphones isplaced inside the ear canal of a real person, or a dummy head, to make arecording. When the recording is played back with headphones orearphones or otherwise manipulated to generate these recorded sounds ata location in the listener close to where the microphones were placed—inor near the ear canal—then the direction cues perceived by the listenerof the original recording are reproduced on playback and the listeneraccurately perceives 3D Sound.

Sounds can be recorded in binaural by using microphones placed inside adummy head. Most sound recording are not made with a dummy head. Thesesound recordings can be transformed into recordings that generate allthe directional cues that would have been present had the recording beenmade with a dummy head. This is a function of the anatomy of the head.This function is called the Head Related Transfer Function (HRTF). As anexample, three important direction cues incorporated into the HRTF arethe interaural time difference (ITD), the interaural level difference(ILD), and the reverberation in the pinna. ITD is the difference inarrival time for a sound at each ear—a sound coming from the left arriveat the left ear slightly before it arrives at the right ear. ILD is thedifference in loudness—a sound coming from the left is slightly louderat the left ear than it is at the right ear, because the head absorbssome of the sound and creates a “sound shadow” which has the right earinside. Reverberation in the pinna refers to the reflection of sound bythe shapes and anatomical features of the pinna, the flap of tissue thatmakes up the visible part of the ear. All of these effects are combinedinto one transformation of a sound into a 3D Sound, and the quality andperformance of this transformation is a subject of competition betweenvarious 3D Sound commercial software vendors.

An exemplary embodiment of a process/method 100 of generating a 3D soundconference is shown in the flow diagram of FIG. 1. The process begins inoperation 102 in which a virtual conference room map 200, an example ofwhich is shown in FIG. 2, is generated in software or updated anddisplayed on a host computer. This map may be called up on a potentialparticipant's computer display screen. Each potential user, i.e. aparticipant “U_(n)” 202, then accesses this map 200 from a remotecomputer connected to the software on the host computer via theInternet.

Once the map 200 is generated and displayed to a plurality of potentialparticipants on their remote displays, each of the potentialparticipants selects a position, such as U_(n) 202, i.e. one of thechairs on the map shown in FIG. 2. Alternatively, each of theparticipants is assigned a position on the map 200. Once the desirednumber of participants have each selected a seat, or been assigned aseat on the map on their display screens, control transfers to operation104.

In operation 104, each participant speaks and the sound picked up fromeach participant when speaking is leveled. This may be done initially byasking the participant to say his or her name, or recite a predeterminedphrase or the like, during a predetermined block of time, and then inoperation 104 is updated automatically for each interval of time. Inaddition to providing consistent volume and verifying microphoneoperation, leveler 104 provides the important function of removing thedirection cues about where the speaking participant is relative to aphysical microphone so that the system can replace those cues withcomputer generated cues about where the speaking participant is in thevirtual room map. When the sound is leveled for each speakingparticipant control transfers to operation 106.

In operation 106, 3D sound is generated for each listening participant.A listening participant is identified as “L_(n)”. More particularly,basic 3D sound generation is explained. Here the sound received inoperation 106 is converted into a converted sound for each listeningparticipant. This converted sound is slightly different for eachposition on the map shown in FIG. 2 according to the direction of eachposition from the position associated with a speaking participant whogenerated the sound received. For example, in a virtual room with 8positions in a circle, the converted sound received from a speaker inposition U1 would be changed differently for sending to each ofpositions U2 through U6, according to the particular direction betweenpositions U1 and U3, U1 and U4, U1 and U5, U1 and U6, etc. The listeningparticipant 204, U4, for example, will perceive the converted sound fromthe speaker in position 205, U5, as if it was coming from his/her left.Similarly a listening participant 204, U4, would perceive the convertedsound from the speaker in position 203, U3, as if it was coming fromhis/her right. Thus the converted sound received from position 206, U1,is converted differently according to the listening participant'sposition direction from the speaking participant. Control then transfersto query operation 108.

Query operation 108 asks whether the software functionality has beenselected to attenuate each speaking participant as a function ofdistance from a speaking participant. If the answer is yes, then controltransfers to operation 110. If the answer in query operation 108 is no,then control transfers to operation 112.

In operation 110, each speaking participant's voice that is to be sentto each listening participant is partially or completely attenuated as afunction of distance from that listening participant to the speakingparticipant. Control then transfers to operation 112. In other words, ifthere is a large table map, participants at the ends of the table willsound further away than participants sitting closer to the listeningparticipant.

In operation 112, from each listening participant L_(n)'s position onthe map 200, the direction of each other participant, i.e., each otherspeaking participant Sn is determined. This information is then stored,for example, in a lookup table in a database associated with theconference, for immediate access. Control then transfers to operation114.

In operation 114, the sound from each speaking participant Ln is mixedtogether with each adjacent speaking participant's sound based on theirrelative positions in the virtual room and their direction from eachadjacent speaking participant. Control then transfers to operation 116.

In operation 116, the mixed sound from each speaking participant in thevirtual room is transformed into binaural audio. Control then transfersto query operation 118. In query operation 118, each listeningparticipant identified on the virtual map is queried whether he or sheis actually wearing headphones.

It is to be understood that this operation may be alternately done outof sequence, for example, as part of sound leveling activity inoperation 104, and this information can just be checked or assumed tohave not changed here. However, for purpose of description, it has beenplaced here. Furthermore, query operation 118 may be implemented everysecond, third or fourth iteration, for example, rather than during eachiteration as herein described.

If the listening participant is wearing headphones, then controltransfers to operation 120 where the sound is queued for transmission toeach listening participant L_(n). Alternatively, if the listeningparticipant is not wearing headphones, control transfers to operation122.

In operation 122, a crosstalk cancellation operation is performed on thebinaural audio signal to each participant L_(n) in order to provide thesame effect with the speakers as is achieved with the headphones.Control then transfers to operation 120 where the binaural sound isqueued for transmission to the listening participant L_(n) andtransmitted automatically thereafter. It is to be understood thattransmission may optionally be done out of sequence, for example, afterquery operation 124, if there are no more participants to be accountedfor. However, for purpose of this description, transmission is describedhere.

When binaural sound is cued, then control transfers to query operation124. Query operation 124 asks whether there are any more participants inthe virtual room in the conference during this particular time block. Ifyes, control transfers in operation 126 back to operation 106 where 3Dsound is generated for the listening next participant L_(n). Thesequence from 106 to 124 is then repeated until there are no morelistening participants in query operation 124. When this occurs, controltransfers to operation 128, where the query is made whether it is theend of the conference. If not, control transfers to operation 130 whichinstructs the program to repeat all operations from 102-128 for the nextblock of time.

This sequence of operations 100 takes place relatively quickly such thatit may be transparent to the participants in the conference. Forexample, the block of time involved in each iterative set in thesequence of operations in process 100 may be is typically in the rangeof 1 to 500 milliseconds.

3D Sound Conferencing can be made to emulate a variety of real andvirtual venues or rooms. A different embodiment is used for small,medium, and large conferences, though the sophisticated features usedfor larger conferences can certainly also be applied to smaller ones,and the methods of smaller conferences can be applied to larger ones.

For small venues, typically those with 1-25 participants such asconference room 200 in FIG. 2, we typically use the no reverberationmethod of FIG. 1. At the initiation of the meeting 101, the conferenceroom map, such as 200, is generated and each of n participants, alsoreferred to users U₀ through U_(n), chooses a seat. Alternatively, eachof the n users may be assigned a seat within the conference room map.Each user U₀ through U_(n) is also referred to as speaking participantS₀ through S_(n) when we are concerned with their speaking function andas listening participant L₀ through L_(n) when we are concerned withtheir listening function.

Next we use sound leveler 104 to level the sound from each speaker.Sound levels change all sounds to a similar volume and there arecommercially available sound levelers, such as the AudioCauldronCompressor Engine from Bit Cauldron Corporation. A sound level istypically used so that one song is not considerably louder than the songbefore or after it. In this case we will be using a sound leveler for adifferent reason: the volume level can tell us how loud someone istalking, but it also tells us how far a speaker is from their physicalmicrophone. For 3D sound conferencing, we intentionally level the soundto remove the information about how far the speaker is from theirphysical microphone so that we can then use an attenuator tointentionally add negative or positive volume information thatcommunicates the distance between the speaker (speaking participant) andthe listener (listening participant) in the mapped room.

Not all speakers have their volume attenuated as a function of distancefor all listeners. Decision 108 shows that we may want to selectiveapply either complete, partial, or no attenuation to a specific speakerfor a specific listener (listening participant). There are severalreasons to do this. First, the attenuation information may do more harmthan good to a person who is hard of hearing and that person willbenefit more from a louder sound than from the distance informationconveyed by volume. We call this feature Hearing Aid Clarity, and thisfeature may be turned on or off by each individual listener. Hearing AidClarity can also be turned on and off by the host/administrator of theconference or meeting.

Second, in situation where there is one instructor, or host, doing themajority of the talking, it may be desirable to make the host's volumesuch that the host appears to be at a very short distance from everyone.All of the other direction cues are still present for the host, and allthe direction cues are still present for all the other speakers(speaking participants), we just make the host sound as if you have afront row seat. When the host voice is made to sound a short distancefrom a speaking participant while otherwise preserving the map, we callthis feature Up Close Instruction. Up Close Instruction may be appliedto more than one host, and may be turned on and off by each individuallistener (listening participant) or maybe turned on and off by thehost/administrator of the conference or meeting. The processes describedabove may be performed in the cloud or much of the calculationprocessing may be pushed to the end user's device such as his or hercomputer, desktop, tablet, computer, smart phone, electronic glasses, orother information processing device.

After managing the volume and distance cue from each speakingparticipant to the listener, i.e. listening participant, the geometry ofconference room map 200 is used in calculator operation 112 to calculatethe direction of the sound from the speaking participant to thelistening participant. Each direction may be expressed as an angle onSound ring 300 in FIG. 3. A Sound ring 300 may be visualized as a ringaround the listening participant, for example, listening participant304, that represents the angle of the sound direction relative to theforward facing angle of the listening participant 304, as indicated bythe position of the listening participant's nose 305 from the overheadview of FIG. 3.

Each sound on sound ring 300 may arrive at an arbitrary angle. Thesounds at arbitrary angles along the sound ring are then mixed into afixed number of adjacent angles where virtual speakers have been placed.There may be a very large number of virtual speakers, such as 720speakers, one every half degree, so that each sound can simplest bemoved to the nearest virtual speaker. It is more common to use a smallernumber of virtual speakers, such as a virtual speaker every five degreesor even five virtual speakers total, as in the popular ITU-R BS 775configuration shown for speakers 306L, 306R, 306C, 306LS and 306LR. If asound lands directly on a virtual speaker it is simply mixed entirelyinto that virtual speaker, such as sound 307 landing on virtual speaker306RS. If a sound lands directly between two speakers it can be mixedevenly into those two speakers, as with sound 302 getting mixed intovirtual speakers 306R and 306RS. If a sound is unevenly betweenspeakers, such as sound 301 part way between speakers 306L and 306LS,the sound can be mixed into the nearest neighbor or mixed proportionallybetween the adjacent virtual speakers, the latter of which is the methodused by mixer 114.

The sound is then transformed from the virtual speakers on the soundring to the sound that would be perceived by human ears in this actualsituation, called binaural sound. The converter operation 116 from soundfor virtual speakers to binaural sound is available from commercialsoftware packages, such as the AudioCauldron Headphone Engine from BitCauldron Corporation.

Binaural sound is intended for headphone listening. Query operation 118checks if the user is wearing headphones. If the user is wearingheadphones then the sound is ready to be sent onward through theprocess. If the user is not wearing headphones but is instead listeningwith external physical speakers, then we must cancel the crosstalkeffect introduced by the physical speakers in order to maintain accuratedirection information. Crosstalk canceller operation 122 uses crosstalkcancellation available from commercial software packages, such as iscurrently available via the AudioCauldron Speaker Engine from BitCauldron Corporation.

The process as described to this point creates the directional sound forone listening participant. The process must be repeated to create thesound for each listening participant. All of this processes the soundfor all listening participants for a single short interval of time, forexample, within 1-500 milliseconds. This process must then be repeatedfor the next short interval of time.

For small venues, typically those with 1-25 participants such as avirtual conference room 200 in FIG. 2, we typically use the noreverberation method of FIG. 1. For medium size venues, typically thosewith 26-100 participants, there are more people sitting closer togetherand it can be helpful to distinguish one speaking participant fromanother by adding the additional differentiating cue of reverberation.Any type of reverberation or lack of reverberation can be applied to anyroom size, and what is described herein is merely exemplary. Thereforethe room of room map 200 can be used for a detailed conceptualdescription of the reverberation cue.

Room map 200 shows direct sound path 201 from speaking participant U₅205 to listening participant U₀ 210. The direct path is not the onlypath sound travels. FIG. 4A shows the reverberant paths, the paths soundtravels via reflections off of surfaces in the room. Room map 400L showsthe same room map as room map 200 and the same direct path 401L as path201. FIG. 4A also illustrates reverberant path 403 off of the left wall,404 off of the right wall, 405 off of the rear wall and 406 off of thefront wall. Sound arrives later on these paths because it has farther totravel. These paths also arrive at the sound ring 407 at differentlocations and directions than direct path 401L.

The two dimensionally calculated (2D) reverberations of room 400L aresufficient to add reverberation cues that are specific to each speakingparticipant. Reverberation can be made to sound more natural when takingheight into account. Height is taken into account in two ways. Firstroom map 400R in FIG. 4B shows that direct path 401R also hasreverberant path 408 off of the table and 409 off of the ceiling.Second, all of the participants no longer need to be at the same height.For example, the host can be standing and all of the other participantscan be sitting. This additional height information does not representall possible three dimensional (3D) information, but is considerablymore information than the two dimensional information, so we refer to itas 2.5D information.

2.5D and 3D calculations introduce a new dimension to the sound ring. Inparticular, it now allows that a sound's direction need not sit on asound ring around a person's head, but could originate from anydirection, making the sound ring a sound sphere. We exclude thedirections that are inside your body and your shoulders from the soundsphere and come up with a portion of a sound sphere, which we call asound helmet. FIG. 6 shows a sound helmet as a set of discrete points601 and 602. Virtual speakers could be placed onto each point, betweenpoints, or on a subset of points. The conference system generateselevation information from the mapping of the room, which may includesloped floors or steps, multiple levels, people and objects of differentheights, and other information that determines the elevation ofparticipants, sounds, and reverberated sounds.

An embodiment could calculate reverberation by placing the sound of eachreverberant path at a different location on the sound ring 407, as isshown in FIGS. 4A and 4B, or a sound helmet. In order to both use 2.5Dand 3D reverberation and maximize the energy of the sound cue at thedirection of the direct path, we simplify the reverberation by movingthe sound of all reverberant paths to the location where the direct pathintersects the sound helmet, shown in FIG. 6, point 603F in front view601 and point 603S in side view 602. These simplified paths are alsoshown in FIGS. 5A and 5B for sound maps 500L and 500R. Simplifiedreverberation can give a slightly different sound to each speakingparticipant and improve the experience of having a group conversation,especially in rooms with 26 to 100 people, but can be performed on roomswith any number of people.

For large venues, typically rooms with more than 100 people, our largevenue embodiment also employs blocks and superblocks. Blocks arecontiguous groups of people. FIG. 8 shows 100 seat hall 800. The personin chair 801 is a member of block 802. Superblocks are contiguous groupsof blocks. For example, the highlighted superblock 880 in FIG. 8 is madeup of block 807 and 808.

For groups with 100 or less people, it is possible for everyone toparticipate in a group conversation. For groups of more than 100 people,it becomes less likely that there are more than a few speakingparticipants engaged in a discussion at once, and it becomes more likelythe participants are in a venue, such as a sports arena, with manyparticipants speaking at once. In sports arena settings, the computepower requirements can be significantly reduced without a noticeablechange in quality through the use of blocks and superblocks. The use ofblocks involves three steps. After dividing the venue into blocks, weignore the blocks for all blocks adjacent to or encompassing thelistening participant and calculate the sound from individual speakingparticipants. Next, for blocks at a reasonable distance, all of thespeaking participants in one block are mixed together into one speakingparticipant, and that one speaking participant is treated with onedirection. Finally, for speaking participant blocks far away blocks canbe mixed into superblocks and the superblock can be treated as onespeaking participant.

By taking the small venue method of FIG. 1, adding reverberation formedium venues, and adding blocks and superblocks for large venues, aflowchart of operations to provide 3D sound conferencing for any venuecan be generated. This process is shown in FIG. 7.

At the initiation operation of the meeting 701, the virtual conferenceroom map, such as 200, is generated. In one embodiment, theadministrator chooses a seat for each of n users. In another embodiment,each of n users, U₀ through U_(n), chooses his or her own virtual seat.Each user U₀ through U_(n) is also referred to as speaking participantS₀ through S_(n) when we are concerned with their speaking function andas listening participant L₀ through L_(n) when we are concerned withtheir listening function.

Next we use sound leveler in operation 702 to level the sound from eachspeaking participant. Sound levels change all sounds to a similar volumeand there are commercially available sound levelers, such as theAudioCauldron Compressor Engine from Bit Cauldron Corporation. A soundlevel is typically used so that one voice, such as a song, is notconsiderably louder than the song before or after it. In this case wewill be using a sound leveler for a different reason: the volume levelcan tell us how loud someone is talking, but it also tells us how far aspeaking participant is from their physical microphone. For 3D soundconferencing, we intentionally level the sound to remove the informationabout how far the speaking participant is from their physical microphoneso that we can then use reverberator operation 704, which also providesattenuation, to intentionally add volume information that communicatesthe distance between the speaking participant and the listeningparticipant in the mapped room. The sound leveling also removes bogusinformation distance cues so that when the sound goes to binaural sound,the Bit Cauldron engines can add proper distance to the sound cues andthe sound cues are not distorted because, for example, one speaker is 5feet from his/her microphone.

Reverberator operation 704 calculates reverberation using the method ofhaving all of the reverberation paths arrive at the same point on thesound helmet, and then assigning all of the sound paths summed togetherto that direction.

An interesting phenomenon happens in large venues, such as thatportrayed in FIG. 8. The speed of sound is approximately 1 foot permillisecond (approximately 1000 ft per second) and network latencies aretypically on the order of 50 ms to 100 ms. Latency offset blockoperation 705 subtracts the expected latency of the network from theactual latency caused by air. For example, if the sound path was 100feet in distance and the expected network latency was about 50 ms, thenoffset block 705 would intentionally add only 50 ms of latency to thepath, and the total delay of added latency plus network latency wouldequal the desired latency of 100 ms.

Next, if the speaking participant's sound being processed is actuallypart of a block or superblock, offset block operation 706 furtherreduced the added latency to offset the time required to compute theblock or superblock.

Not all speaking participants have their speaker volume attenuated as afunction of distance for all listening participants. Adjustmentoperation 707 shows that we may want to selective apply either complete,partial, or no attenuation to a specific speaker for a specificparticipant listener. There are several reasons to do this. First, theattenuation information may do more harm than good to a person who ishard of hearing and will benefit more from a louder sound than from thedistance information conveyed by volume. We call this feature HearingAid Clarity, and this feature may be turned on or off.

Second, in a situation where there is one instructor participant or hostin the conference doing the majority of the talking, it may be desirableto make the host's volume such that the host appears to be at a veryshort distance from everyone. All of the other direction cues are stillpresent for the host, and all the direction cues are still present forall the other speaking participants, we just make the host participantsound as if the listening participant has a front row seat. When thehost voice is made to sound a short distance from a speaking participantwhile otherwise preserving the map, we call this feature Up CloseInstruction. Up Close Instruction may be applied to more than onespeaking participant, and may be turned on and off.

Along with managing the volume and distance cue from each speakingparticipant to the listening participant, the geometry of conferenceroom map 200 is used in calculator operation 708 to calculate thedirection of the sound from the speaking participant to the listeningparticipant. In FIG. 1, each direction was expressed as an angle onSound-ring 300 in FIG. 3. Here, each direction is expressed as an angleand elevation in a sound helmet, as is shown in FIG. 6.

Each sound on the sound helmet may arrive at an arbitrary angle andelevation. The sounds at arbitrary angles along the sound ring are thenmixed into a fixed number of positions on the sound helmet where virtualspeakers have been placed. There may be a very large number of virtualspeakers, such as 720 speakers, so that each sound can simply be movedto the nearest virtual speaker. It is more common to use a smallernumber of virtual speakers, such as 11, 18 or 22 speakers arranged invarious configurations that spread the virtual speakers around the soundhelmet. If a sound lands directly on a virtual speaker it is simplymixed entirely into that virtual speaker. If a sound lands directlybetween two speakers it can be mixed proportionally between those twospeakers. In the general case, a sound direction will be at an arbitrarypoint in the curved surface of the sound-helmet and will be mixedproportionally into the four surrounding speakers, which is the methodused by mixer operation 709.

The sound is then transformed from the virtual speakers on thesound-helmet to the sound that would be perceived by human ears in thisactual situation, called binaural sound. The converter operation 710from virtual speakers to binaural sound is available from commercialsoftware packages, such as the AudioCauldron Headphone Engine from BitCauldron Corporation. Control then transfers to query operation 711.

Binaural sound is intended for headphone listening. Query operation 711checks if the user is wearing headphones. If the user is wearingheadphones then the sound is ready to be sent onward through theprocess, and the binaural sound is queued for transmission to thelistening participant L_(N) and may be automatically transmittedthereafter.

If the user is not wearing headphones but is instead listening withphysical speakers, i.e., the answer in query operation is NO, then wemust cancel the crosstalk effect introduced by speakers. Controltherefore transfers to operation 712. Crosstalk canceller operation 712uses crosstalk cancellation available from commercial software packages,such as the AudioCauldron Speaker Engine from Bit Cauldron Corporation.

The binaural sound thus generated is then queued for transmission andtransmitted to the listening participant L_(N). It is to be understoodthat transmission to the listening participant L_(N) may be done out ofsequence, for example, after query of more participants, if there are nomore participants to be accounted for. However, for the purposes of thisdescription, transmission is described here.

The process as described to this point creates the direction sound forone listener. The process must be repeated to create the sound for eachlistener. All of this processes the sound for all listeners for a singleshort interval of time. This process must then be repeated for the nextshort interval of time. Typical short intervals of time are in the 1 to500 millisecond range, such as 9, 10, or 11 milliseconds. The processchecks for more participants, and then checks to see if the conferenceis still going. If so, the process repeats for the next interval oftime. The processes described above may be performed in the cloud ormuch of the calculation processing may be pushed to the end user'sdevice such as his or her computer, desktop, tablet, computer, smartphone, electronic glasses, or other information processing device.

From this description, it will be appreciated that certain aspects areembodied in the user devices, certain aspects are embodied in the serversystems, and certain aspects are embodied in a client/server system as awhole. Embodiments disclosed can be implemented using hardware, programsof instruction, or combinations of hardware and programs ofinstructions.

In general, routines executed to implement the embodiments may beimplemented as part of an operating system or a specific application,component, program, object, module or sequence of instructions referredto as “computer programs.” The computer programs typically comprise oneor more instructions set at various times in various memory and storagedevices in a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

While some embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that various embodiments are capable of beingdistributed as a program product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

Examples of computer-readable media include but are not limited torecordable and non-recordable type media such as volatile andnon-volatile memory devices, read only memory (ROM), or random accessmemory. In this description, various functions and operations aredescribed as being performed by or caused by software code to simplifydescription. However, those skilled in the art will recognize what ismeant by such expressions is that the functions result from execution ofthe code by a processor, such as a microprocessor.

FIG. 9 shows one example of a schematic diagram illustrating a clientdevice 905 upon which an exemplary embodiment of the present disclosuremay be implemented. Client device 905 may include a computing devicecapable of sending or receiving signals, such as via a wired or wirelessnetwork. A client device 905 may, for example, include a desktopcomputer or a portable device, such as a cellular telephone, asmartphone, a display pager, a radio frequency (RF) device, an infrared(IR) device, a Personal Digital Assistant (PDA), augmented realityglasses, a handheld computer, a tablet computer, a laptop computer, adigital camera, a set top box, a wearable computer, an integrated devicecombining various features, such as features of the foregoing devices,or the like.

The client device 905 may vary in terms of capabilities or features.Claimed subject matter is intended to cover a wide range of potentialvariations. For example, a cell phone may include a numeric keypad or adisplay of limited functionality, such as a monochrome liquid crystaldisplay (LCD) for displaying text, pictures, etc. In contrast, however,as another example, a web-enabled client device may include one or morephysical or virtual keyboards, mass storage, one or more accelerometers,one or more gyroscopes, global positioning system (GPS) or otherlocation-identifying type capability, of a display with a high degree offunctionality, such as a touch-sensitive color 2D or 3D display, forexample. Other examples included augmented reality glasses and tablets.

A client device 905 may include or may execute a variety of operatingsystems, including a personal computer operating system, such as aWindows, iOS or Linux, or a mobile operating system, such as iOS,Android, or Windows Mobile, or the like. A client device may include ormay execute a variety of possible applications, such as a clientsoftware application enabling communication with other devices, such ascommunicating one or more messages, such as via email, short messageservice (SMS), or multimedia message service (MMS), including via anetwork, such as a social network, including, for example, Facebook®,LinkedIn®, Twitter®, Flickr®, or Google+®, to provide only a fewpossible examples. A client device may also include or execute anapplication to communicate content, such as, for example, textualcontent, multimedia content, or the like. A client device may alsoinclude or execute an application to perform a variety of possibletasks, such as browsing, searching, playing various forms of content,including locally stored or streamed video, or games (such as fantasysports leagues). The foregoing is provided to illustrate that claimedsubject matter is intended to include a wide range of possible featuresor capabilities.

As shown in the example of FIG. 9, client device 905 may include one ormore processing units (also referred to herein as CPUs) 922, whichinterface with at least one computer bus 925. A memory 930 can bepersistent storage and interfaces with the computer bus 925. The memory930 includes RAM 932 and ROM 934. ROM 934 includes a BIOS 940. Memory930 interfaces with computer bus 925 so as to provide information storedin memory 930 to CPU 922 during execution of software programs such asan operating system 941, application programs 942 such as device drivers(not shown), and software messenger module 943 and browser module 945,that comprise program code, and/or computer-executable process steps,incorporating functionality described herein, e.g., one or more ofprocess flows described herein. CPU 922 first loads computer-executableprocess steps from storage, e.g., memory 932, data storage medium/media944, removable media drive, and/or other storage device. CPU 922 canthen execute the stored process steps in order to execute the loadedcomputer-executable process steps. Stored data, e.g., data stored by astorage device, can be accessed by CPU 922 during the execution ofcomputer-executable process steps.

Persistent storage medium/media 944 is a computer readable storagemedium(s) that can be used to store software and data, e.g., anoperating system and one or more application programs. Persistentstorage medium/media 944 can also be used to store device drivers, suchas one or more of a digital camera driver, monitor driver, printerdriver, scanner driver, or other device drivers, web pages, contentfiles, playlists and other files. Persistent storage medium/media 906can further include program modules and data files used to implement oneor more embodiments of the present disclosure.

For the purposes of this disclosure a computer readable medium storescomputer data, which data can include computer program code that isexecutable by a computer, in machine readable form. By way of example,and not limitation, a computer readable medium may comprise computerreadable storage media, for tangible or fixed storage of data, orcommunication media for transient interpretation of code-containingsignals. Computer readable storage media, as used herein, refers tophysical or tangible storage (as opposed to signals) and includeswithout limitation volatile and non-volatile, removable andnon-removable media implemented in any method or technology for thetangible storage of information such as computer-readable instructions,data structures, program modules or other data. Computer readablestorage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM,flash memory or other solid state memory technology, CD-ROM, DVD, orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other physical ormaterial medium which can be used to tangibly store the desiredinformation or data or instructions and which can be accessed by acomputer or processor.

Client device 905 can also include one or more of a power supply 926,network interface 950, audio interface 952, a display 954 (e.g., amonitor or screen), keypad 956, illuminator 958, I/O interface 960, ahaptic interface 962, a GPS 964, and/or a microphone 966.

For the purposes of this disclosure a module is a software, hardware, orfirmware (or combinations thereof) system, process or functionality, orcomponent thereof, that performs or facilitates the processes, features,and/or functions described herein (with or without human interaction oraugmentation). A module can include sub-modules. Software components ofa module may be stored on a computer readable medium. Modules may beintegral to one or more servers, or be loaded and executed by one ormore servers. One or more modules may be grouped into an engine or anapplication.

FIG. 10 is a block diagram illustrating an internal architecture 1000 ofan example of a computer, such as server computer and/or client device,in accordance with one or more embodiments of the present disclosure. Acomputer as referred to herein refers to any device with a processorcapable of executing logic or coded instructions, and could be a server,personal computer, set top box, tablet, smart phone, pad computer ormedia device, or augmented reality glasses, to name a few such devices.As shown in the example of FIG. 10, internal architecture 1000 includesone or more processing units (also referred to herein as CPUs) 1012,which interface with at least one computer bus 1002. Also interfacingwith computer bus 1002 are persistent storage medium/media 1006, networkinterface 1014, memory 1004, e.g., random access memory (RAM), run-timetransient memory, read only memory (ROM), etc., media disk driveinterface 1008 as an interface for a drive that can read and/or write tomedia including removable media such as floppy, CD-ROM, DVD, etc. media,display interface 1010 as interface for a monitor or other displaydevice, keyboard interface 1016 as interface for a keyboard, pointingdevice interface 1018 as an interface for a mouse or other pointingdevice, CD/DVD drive interface 1020, and miscellaneous other interfaces1022, such as parallel and serial port interfaces, a universal serialbus (USB) interface, Apple's ThunderBolt and Firewire port interfaces,and the like.

Memory 1004 interfaces with computer bus 1002 so as to provideinformation stored in memory 1004 to CPU 1012 during execution ofsoftware programs such as an operating system, application programs,device drivers, and software modules that comprise program code, and/orcomputer-executable process steps, incorporating functionality describedherein, e.g., one or more of process flows described herein. CPU 1012first loads computer-executable process steps from storage, e.g., memory1004, storage medium/media 1006, removable media drive, and/or otherstorage device. CPU 1012 can then execute the stored process steps inorder to execute the loaded computer-executable process steps. Storeddata, e.g., data stored by a storage device, can be accessed by CPU 1012during the execution of computer-executable process steps.

As described above, persistent storage medium/media 1006 is a computerreadable storage medium(s) that can be used to store software and data,e.g., an operating system and one or more application programs.Persistent storage medium/media 1006 can also be used to store devicedrivers, such as one or more of a digital camera driver, monitor driver,printer driver, scanner driver, or other device drivers, web pages,content files, playlists and other files. Persistent storagemedium/media 1006 can further include program modules and data filesused to implement one or more embodiments of the present disclosure.

For the purposes of this disclosure a module is a software, hardware, orfirmware (or combinations thereof) system, process or functionality, orcomponent thereof, that performs or facilitates the processes, features,and/or functions described herein (with or without human interaction oraugmentation). A module can include sub-modules. Software components ofa module may be stored on a computer readable medium. Modules may beintegral to one or more servers, or be loaded and executed by one ormore servers. One or more modules may be grouped into an engine or anapplication.

FIG. 10 is a block diagram illustrating an internal architecture 1000 ofan example of a computer, such as server computer and/or client device,in accordance with one or more embodiments of the present disclosure. Acomputer as referred to herein refers to any device with a processorcapable of executing logic or coded instructions, and could be a server,personal computer, set top box, tablet, smart phone, pad computer ormedia device, or augmented reality glasses, to name a few such devices.As shown in the example of FIG. 10, internal architecture 1000 includesone or more processing units (also referred to herein as CPUs) 1012,which interface with at least one computer bus 1002. Also interfacingwith computer bus 1002 are persistent storage medium/media 1006, networkinterface 1014, memory 1004, e.g., random access memory (RAM), run-timetransient memory, read only memory (ROM), etc., media disk driveinterface 1008 as an interface for a drive that can read and/or write tomedia including removable media such as floppy, CD-ROM, DVD, etc. media,display interface 1010 as interface for a monitor or other displaydevice, keyboard interface 1016 as interface for a keyboard, pointingdevice interface 1018 as an interface for a mouse or other pointingdevice, CD/DVD drive interface 1020, and miscellaneous other interfaces1022, such as parallel and serial port interfaces, a universal serialbus (USB) interface, Apple's ThunderBolt and Firewire port interfaces,and the like.

Memory 1004 interfaces with computer bus 1002 so as to provideinformation stored in memory 1004 to CPU 1012 during execution ofsoftware programs such as an operating system, application programs,device drivers, and software modules that comprise program code, and/orcomputer-executable process steps, incorporating functionality describedherein, e.g., one or more of process flows described herein. CPU 1012first loads computer-executable process steps from storage, e.g., memory1004, storage medium/media 1006, removable media drive, and/or otherstorage device. CPU 1012 can then execute the stored process steps inorder to execute the loaded computer-executable process steps. Storeddata, e.g., data stored by a storage device, can be accessed by CPU 1012during the execution of computer-executable process steps.

As described above, persistent storage medium/media 1006 is a computerreadable storage medium(s) that can be used to store software and data,e.g., an operating system and one or more application programs.Persistent storage medium/media 1006 can also be used to store devicedrivers, such as one or more of a digital camera driver, monitor driver,printer driver, scanner driver, or other device drivers, web pages,content files, playlists and other files. Persistent storagemedium/media 1006 can further include program modules and data filesused to implement one or more embodiments of the present disclosure.

Another embodiment in accordance with the present disclosure is shownwith reference to FIGS. 11-14. This embodiment is particularly suitedfor use with television broadcasts. Many meetings, conferences, anddiscussions take place in which there is overlap between speakers—morethan one person is talking at a time. In a real life panel discussion1100 as shown in FIG. 11a , people hear direction cues from the actualspeakers that help the listener understand one speaker even when anotherspeaker is talking at the same time. However, when the sound is capturedby microphones for amplification or recording during a televisionbroadcast, these direction cues are typically lost.

To restore the ability to hear overlapping speakers in accordance withthis embodiment of the present disclosure, a microphone andcorresponding speaker is placed with each of the participants and arereflected in a virtual room like the one in FIG. 11b . Direction cuesfor each speaking participant and listening participant are provided ashas been described above, are based on their position in the virtualroom, and the audio output combined in accordance with the presentdisclosure, converted to binaural sound, and broadcast or otherwisetransmitted to a television viewer. Thus the television viewer can havea more optimum listening experience.

FIG. 12 shows a mixer configuration with 3D Sound Positioners for apanel discussion to be televised in accordance with the presentdisclosure. In this embodiment, a 3D Sound positioner is added to eachchannel of a conventional sound mixer with each participant H and G1-G4given a microphone 1201, 1203, 1205, 1207, 1209. Each microphone feeds aseparate one of the mixer channels 1202, 1204, 1206, 1208, and 1210.Thus channel 1202 for the host microphone H (1201) has a 3D soundpositioner. Channel 1204 for Mic. G1 (1203) has its own 3D Soundpositioner 1221, etc. Each of the 3D sound positioners feed into asumming amplifier 1218 for output.

In the current embodiment shown in FIGS. 11-14, a 3D Sound positioner isadded to the mixing process as shown in FIG. 12. 3D Sound Positioner1220 takes the sound of 1101 host speaker H and transforms it viamicrophone 1201, its mixer channel 1202 and its 3D positioner 1220 suchthat cues regarding it's perceived direction are reintroduced.

In particular, the sound in each channel has been transformed intobinaural sound, i.e., sound which has a set of cues that place it at aparticular position, both azimuth and elevation, relative to the forwardlooking direction of a virtual listener. This positioning can still bedone by an automated or manual sliding control as illustrated, whereinthe sliding control now determines an angle relative to a hypotheticallistener that is approximately in the plane created by the listener'snose and ears, at a location representative of a typical televisionviewer from the television. The positioning can alternatively bedetermined by a set of two automated or manual sliding controls (notshown), wherein one determines an angle relative to a virtual listenerthat is approximately in the plane created by the hypotheticallistener's nose and ear, and one that controls an elevation. Thispositioning can also be determined by selecting a point on a circle thatrepresents a set of possible directions around the listener's head. Thepositioning can also be accomplished by selecting a point on or in acircle, where the distance inside the circle represents an elevationover or under the plane formed by the virtual listener's nose and ears.

The mixing process as shown in FIG. 13. 3D Sound Positioner 1220 takesthe sound of host speaker H (1101) and transforms it such that cuesregarding it's perceived direction have been reintroduced. Inparticular, the sound has been transformed into binaural sound, soundwhich has a set of cues that place it at a particular position, bothazimuth and elevation, relative to the forward looking direction of thelistener.

Television viewers sometimes prefer to view conference participants aspanels of images tiled onto a screen, as panels 1108, 1109, 1110, 1111and 1112 on screen 1107 in FIG. 11c . This configuration makes itstraightforward for the viewer to see all of the participants at thesame time. This configuration makes it easy to hear all of theparticipants at the same time by arranging their audio direction in thesame manner as their images have been arranged.

Also, the positioning can be determined by selecting a point on one ormore representation of a sphere or partial sphere of locations aroundthe listener's head. When working with a graphical environment as iscommon in television production, sometimes a set of templates isavailable that allow operators in a television studio to quickly tilemultiple video streams onto a single display as in FIG. 11c . In thisenvironment, a map with the same configuration as the video template, oran enhancement to the video template itself, can be used to provide a 3DSound Positioner for the audio associated with each video stream. FIG.14 shows video template 1401 enhanced with a 3D Sound Positioner in thebottom of each video tile 1402, 1403, 1404, 1405, and 1406.

An exemplary workflow for including 3D Sound Positioning in a televisionproduction sequence is shown in FIG. 13. After the start process inoperation 1300, control transfers to operation 1301. Operation 1301generates a virtual meeting room tiled window map that matches thepositions of all of the tiles in the template. Control then transfers tooperation 1302. Operation 1302 adds a 3D Sound Positioner and UserInterface to each tile. Because speakers sound different based on theirposition relative to the listener, each speaker sounds different as thespeaker's sound direction is changed. To avoid the abrupt change insound that may accompany an abrupt change in position, the speakerssounds can be positioned in their 3D Sound Positions before thebeginning of the program or a ramp can be provided to limit the speed atwhich the direction of a speaker can change. Control then transfers toquery operation 1303. Decision 1303, or query operation 1303, checks ifwe are in a state of transition and if so control transfers to operation1304. Process operation 1304 changes the old position incrementally soit gets closer to a new desired position. Then control transfers toprocess operation 1306 determined the incrementally changed positions tobe used for this block of time. Process 1307 mixes all of the soundsinto a fixed number of virtual speaker positions on a sound helmet.Commercial Software process operation 1308 transforms the sounds fromtheir positions on the sound helmet into binaural audio and the outputis queued for transmission to speakers, television broadcast, or arecording medium, and may be automatically transmitted thereafter to aviewer.

Those skilled in the art will recognize that the methods and systems ofthe present disclosure may be implemented in many manners and as suchare not to be limited by the foregoing exemplary embodiments andexamples. In other words, functional elements being performed by singleor multiple components, in various combinations of hardware and softwareor firmware, and individual functions, may be distributed among softwareapplications at either the user computing device or server or both. Inthis regard, any number of the features of the different embodimentsdescribed herein may be combined into single or multiple embodiments,and alternate embodiments having fewer than, or more than, all of thefeatures described herein are possible. Functionality may also be, inwhole or in part, distributed among multiple components, in manners nowknown or to become known. Thus, myriad software/hardware/firmwarecombinations are possible in achieving the functions, features,interfaces and preferences described herein. Moreover, the scope of thepresent disclosure covers conventionally known manners for carrying outthe described features and functions and interfaces, as well as thosevariations and modifications that may be made to the hardware orsoftware or firmware components described herein as would be understoodby those skilled in the art now and hereafter.

Although some of the drawings illustrate a number of operations in aparticular order, operations which are not order dependent may bereordered and other operations may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beapparent to those of ordinary skill in the art and so do not present anexhaustive list of alternatives. Moreover, it should be recognized thatthe stages could be implemented in hardware, firmware, software or anycombination thereof.

Although the disclosure has been provided with reference to specificexemplary embodiments, it will be evident that the various modificationand changes can be made to these embodiments without departing from thebroader spirit as set forth in the claims. For example, provision couldbe made for additional listening participants beyond the number ofchairs in the virtual room. In such case, these listening participantswould hear as if they were either in a predetermined one of thepositions, or without the benefit of 3D sound. Accordingly, thespecification and drawings are to be regarded in an illustrative senserather than in a restrictive sense. All such changes, alternatives andequivalents in accordance with the features and benefits describedherein, are within the scope of the present disclosure. Such changes andalternatives may be introduced without departing from the spirit andbroad scope of my invention as defined by the claims below and theirequivalents.

What is claimed is:
 1. A computer implemented method of audioprogramming for a plurality of participants in a televised discussionbetween the participants, the method comprising: assigning, via aprocessor, each of the plurality of participants to a different positionon a computer generated map, wherein the plurality of participantsincludes speaking participants and listening participants; determining adirection on the map of each speaking participant from a listeningparticipant; receiving a voice sound from a speaking participant;leveling the received voice sound from the speaking participant to acommon leveled voice sound for the speaking participant; attenuating theleveled voice sound to an attenuated voice sound, such that theattenuated sound is representative of a distance between the speakingparticipant and the listening participants; converting the leveled andattenuated voice sound to a converted sound corresponding to thedirection to each of the listening participants from the speakingparticipant; and transforming the converted sound to binaural sound foraudio transmission.
 2. A method according to claim 1 wherein the map isa virtual room.
 3. A method according to claim 2 wherein the virtualroom has a plurality of predefined unique positions for the speakingparticipants.
 4. A method according to claim 3 wherein each listeningparticipant is at a predetermined position within the virtual room. 5.The method of claim 1 wherein the sound from each speaking participantis assigned to a virtual speaker associated with a direction from eachlistening participant.
 6. A method according to claim 1 wherein the mapincludes a sound ring centered around each listening participant.
 7. Themethod according to claim 6 wherein each virtual speaker is associatedwith a position around the sound ring.
 8. The method according to claim2 further comprising the virtual room having a plurality of wallsdefining the room.
 9. The method according to claim 8 whereinreverberation is added to the received sound directed to each of thevirtual speakers other than the virtual speaker associated with thespeaking participant.
 10. The method according to claim 1 wherein thevoice sound from each speaking participant is converted to a modifiedvoice.
 11. The method according to claim 10 wherein each modified voiceis uniquely associated with its position in a virtual room.
 12. Themethod according to claim 1 further comprising compensating theconverted sound for an elevation difference between the speakingparticipant and an elevation of the listener.
 13. The method accordingto claim 8 further comprising compensating the converted sound for anelevation difference between the speaking participant and an elevationof the listener.
 14. The method according to claim 12 wherein theelevation compensation of the converted sound is adjusted in anincremental manner so as not to be disorienting to a viewer.
 15. Anon-transient non-transitory tangible machine readable storage medium,storing instructions that, when executed by a computing device, causethe computing device to perform a method of audio programming for aplurality of participants in a televised discussion, the methodcomprising: assigning, by the computing device, the plurality ofparticipants each to a different position on a map, wherein theplurality of participants includes speaking participants and listeningparticipants; determining, by the computing device, a direction on themap of each speaking participant from each listening participant;receiving, by the computing device, a voice sound from a speakingparticipant; leveling, by the computing device, the voice sound from thespeaking participant to a common leveled voice sound for the speakingparticipant; attenuating, by the computing device, a volume of theleveled voice sound to an attenuated sound, such that the attenuatedsound is representative of a distance between the speaking participantand each of the listening participants; converting, by the computingdevice, the leveled and attenuated voice sound to a converted soundcorresponding to the direction to the listening participants from thespeaking participant; and transforming, by the computing device, theconverted sound to binaural sound for audio transmission of thetelevised discussion.
 16. The medium according to claim 15 wherein themap is a virtual room.
 17. The medium according to claim 16 wherein thevirtual room has a plurality of predefined unique positions.
 18. Acomputer system comprising: a memory storing instructions; and aprocessor coupled with the memory to execute the instructions, theinstructions configured to instruct the processor to: assign to each ofa plurality of participants at a televised discussion a differentposition on a computer generated map, wherein the plurality ofparticipants includes speaking participants and listening participants;determine a direction on the map of each speaking participant from eachlistening participant; receive a voice sound from a speakingparticipant; level the voice sound from the speaking participant to acommon leveled voice sound for the speaking participant; attenuate avolume of the leveled voice sound to an attenuated sound, such that theattenuated sound is representative of a distance between the speakingparticipant and each of the listening participants; convert the leveledand attenuated voice sound to a converted sound corresponding to thedirection to the listening participants from the speaking participant;and transform the converted sound to binaural sound for audiotransmission of the televised discussion.
 19. The system according toclaim 18 wherein the map is a virtual room.
 20. The system according toclaim 19 wherein the virtual room has a plurality of predefined uniquepositions.
 21. A computer implemented method of audio programming for aplurality of participants in a televised discussion between theparticipants, the method comprising: assigning, via a processor, each ofthe plurality of participants to a different position on a computergenerated map, wherein the plurality of participants includes speakingparticipants and listening participants; determining a direction on themap of each speaking participant from each listening participant;receiving a voice sound from a speaking participant; leveling the voicesound from the speaking participant to a common leveled voice sound forthe speaking participant; converting the leveled voice sound to aconverted sound corresponding to the direction to each of the listeningparticipants from the speaking participant; attenuating the leveled andconverted voice sound to an attenuated voice sound, such that theattenuated sound is representative of a distance between the speakingparticipant and the listening participants; and transforming theattenuated sound to binaural sound for audio transmission of thetelevised discussion.
 22. A non-transient non-transitory tangiblemachine readable storage medium, storing instructions that, whenexecuted by a computing device, cause the computing device to perform amethod of audio programming for a plurality of participants in atelevised discussion, the method comprising: assigning, by the computingdevice, the plurality of participants each to a different position on amap, wherein the plurality of participants includes speakingparticipants and listening participants; determining, by the computingdevice, a direction on the map of each speaking participant from eachlistening participant; receiving, by the computing device, a voice soundfrom a speaking participant; leveling, by the computing device, thevoice sound from the speaking participant to a common leveled voicesound for the speaking participant; converting, by the computing device,the leveled voice sound to a converted sound corresponding to thedirection to the listening participants from the speaking participant;attenuating, by the computing device, a volume of the leveled andconverted voice sound to an attenuated sound, such that the attenuatedsound is representative of a distance between the speaking participantand each of the listening participants; and transforming, by thecomputing device, the attenuated sound to binaural sound for audiotransmission of the televised discussion.
 23. A computer systemcomprising: a memory storing instructions; and a processor coupled withthe memory to execute the instructions, the instructions configured toinstruct the processor to: assign to each of a plurality of participantsat a televised discussion a different position on a computer generatedmap, wherein the plurality of participants includes speakingparticipants and listening participants; determine a direction on themap of each speaking participant from each listening participant;receive a voice sound from a speaking participant; level the voice soundfrom the speaking participant to a common leveled voice sound for thespeaking participant; convert the leveled voice sound to a convertedsound corresponding to the direction to the listening participants fromthe speaking participant; attenuate a volume of the leveled andconverted voice sound to an attenuated sound, such that the attenuatedsound is representative of a distance between the speaking participantand each of the listening participants; and transform the attenuatedsound to binaural sound for audio transmission of the televiseddiscussion.